Large transformer models have achieved state-of-the-art results in numerous natural language processing tasks. Among the pivotal components of the transformer architecture, the attention mechanism plays a crucial role in capturing token interactions within sequences through the utilization of softmax function. Conversely, linear attention presents a more computationally efficient alternative by approximating the softmax operation with linear complexity. However, it exhibits substantial performance degradation when compared to the traditional softmax attention mechanism. In this paper, we bridge the gap in our theoretical understanding of the reasons behind the practical performance gap between softmax and linear attention. By conducting a comprehensive comparative analysis of these two attention mechanisms, we shed light on the underlying reasons for why softmax attention outperforms linear attention in most scenarios.

Expeditionary Artificial Intelligence is the theme for the May 2020 research and experimentation week during which Private Industry, Academia, and Non-Government Organizations will collaborate on developing and demonstrating Machine and Deep Learning, sensor, networked all-domain Autonomous Systems, and other technologies. The May 2020 experiment will be co-hosted by The Sea Land Air Military Research initiative (SLAMR) and Joint Interagency Field Experimentation (JIFX) Program. The initial TechOp Day at the Naval Postgraduate School (NPS) was part of the laying the groundwork with Private Industry and other partners. This video was produced by NPS and can be watched at NPS' YouTube channel at https://www.youtube.com/user/NPSvideo.